Method and apparatus for representing and editing multimedia compositions using references to tracks in the composition to define components of the composition

ABSTRACT

A system for editing and representing multimedia compositions in which the representation of a composition enables a composition to be an arbitrarily deep nesting of assemblies of components. In this representation, relationships between synchronous components may be represented, by an array or list. Components which are related sequentially in time are represented by a sequence component, which may be implemented as a tree. For the purpose of improving searching within the tree, a sequence component is implemented as a balanced binary, or 2-3-tree. By using such a representation, searching for a component based on its position in time in the sequence may be performed in logarithmic rather than linear time. Each node in the tree has a corresponding duration, which represents the sum of the durations of any subnodes. Media data for a composition is excluded from the representational structures and is merely referred to by such structures. Thus, compositions may be stored separately from the media data.

This application is a division of application Ser. No. 08/415,091, filedon Mar. 31, 1995, pending, which is a file-wrapper continuationapplication of U.S. application Ser. No. 08/236,244, filed Apr. 21,1994, abandoned, which is a file-wrapper continuation application ofU.S. application Ser. No. 07/920,260, filed Jul. 27, 1992, abandoned,entitled METHOD AND APPARATUS FOR REPRESENTING AND EDITING MULTIMEDIACOMPOSITIONS, which is a continuation-in-part of application Ser. No.07/867,133, filed Apr. 10, 1992, abandoned, entitled COMPUTER SYSTEM FORVIDEO EDITING AND POST-PRODUCTION.

FIELD OF THE INVENTION

This invention is related to methods and apparatus for editing andrepresenting multimedia compositions. More particularly, the inventionis related to video and audio editing systems for creating videoprograms, and for representing such video programs.

BACKGROUND OF THE INVENTION

Known representations of relationships of video and audio informationthat comprise a video program are very limited. Edit decision lists(EDL) have been used throughout the video industry. An EDL consists of asequence of event descriptions, where each event is a transfer of aprogram segment from a source tape to a master tape. There are a numberof available EDL formats, but each format conveys similar information.The event description contains such information as the source of a newprogram segment, time codes describing both a desired portion of thesource and its destination in the final program, and the type of editedtransition that is to be used from any previous segment. From thisinformation, the final program is constructed from the several sourcesof program material. EDL systems are limited in their representationalcapability and content, and are inflexible. Further, because an EDL islimited in its content, and since there are many different EDL formats,different EDL systems are often incompatible. That is, a video programdeveloped using one EDL format may not be usable by another EDL baseproduct.

More recently, developments have been made in the field of computerizedmulti-media editing systems. Many of these systems use a subsystem,called QuickTime, made by Apple Computer, of Cupertino, Calif. All ofthese systems, however, provide a "flat" representation of a multimediacomposition. That is, the representation is merely linear with time. Inorder to access a location in a composition, a linear search isrequired.

Further, such systems usually have a bare minimum of recordkeeping bynot keeping track of editing steps made to generate the composition. Afinal copy of a composition is generated with these systems, and thereis no record kept of the layering of media in a frame of thecomposition. Without this information, many steps may need to beperformed if an edit is changed.

For example, when editing, analog sources are run in parallel inaccordance with the editing steps and the resulting composition isrecorded on a master tape. Similar steps are used when editing withinformation in digital form as well. In prior systems, the compositionrules and steps are not saved, and thus when an editor wants to changesomething in the final copy, the whole master tape has to be re-recordedto obtain the new composition. Such editing steps are wasteful of timeand materials.

Accordingly, it is a general aim of the present invention to provide amethod and apparatus for representing and editing multimediacompositions which separates the structure of a composition from themedia which it uses.

Further, the present invention was developed to maintain arepresentation of a composition from which layering of and relationshipsbetween media in the composition may be determined.

SUMMARY OF THE INVENTION

The foregoing and other objects of the invention are obtained by asystem for editing and representing multimedia compositions in whichdifferent types of relationships between media are represented bydifferent types of components, and in which components may alsorepresent relationships between different components. A compositionutilizing this representation has a hierarchical, tree-like structure.

In this representation, relationships between synchronous components maybe represented, for example, by an array or list, which can beenvisioned as a tree with one root node and a number of leavescorresponding to the length of the array.

Components which are related sequentially in time are represented by asequence component, which may be implemented as a tree. Although thistree may be considered as a single level with a number of leaves equalto the number of components in the sequence, it is preferable, for thepurpose of searching the tree to implement a sequence component as abalanced binary, or 2-3-tree. By using such a representation, searchingfor a component based on its position in time in the sequence may beperformed in logarithmic, O(logn), rather than linear, O(n), time. In apreferred embodiment, each node in the tree has a correspondingduration, which represents the sum of the durations of any subnodes.

Such a representation of sequential and synchronous components embodiesthe idea that a composition is not just a linear assembly of media, butan assembly of assemblies. Thus, a composition may be an arbitrarilydeep, hierarchical structure.

Media data for a composition is excluded from the representationalstructures and is merely referred to by such structures. Thus,compositions may be stored separately from the media data.

A system in accordance with the invention may also provide operations oncomponents which allow editing, through creation, modification anddeletion of components. Using these operations in conjunction with acomputer system, an editor of multimedia compositions can perform suchfunctions as replacing, overwriting, inserting, extracting and liftingcomponents in a composition.

Multimedia data manipulated by the system of the invention may be mediadata files on a hard disk on a computer system. These media files arepreferably created from original media sources, for example, throughvideo compression or from analog audio sources. References to media datafiles in a composition also include an indication of the original sourcematerial, if any, which enables computer based digital media to be usedwhile constructing a composition, while enabling original sourcematerial to be used when making a final production of the composition.The media data are not part of the composition; the composition onlyrefers to the media.

Compositions created with this editing and representation system may beused by other systems to create edit decision lists, to play thecomposition, to eliminate unnecessary data files, etc. Other advantagesand applications of the present invention should become apparent tothose of skill in this art after a reading of the following descriptionwhich should be read in conjunction with the attached drawing.

BRIEF DESCRIPTION OF THE DRAWING

In the drawing, FIG. 1 is a block diagram describing a computer systemsuitable for implementing the present invention;

FIG. 2 is a diagram of the class hierarchy of object classes of anembodiment of the invention;

FIG. 3 is a diagram illustrating the hierarchy of a composition;

FIG. 4 is a diagram illustrating a suitable user interface for use withthe present invention;

FIG. 5 is a flow chart describing operations on source and targetmaterials;

FIG. 6 is a flow chart describing operations on target materials;

FIG. 7 is a flow chart describing a function for ending an editoperation;

FIG. 8 is a flow chart describing how an edit description is propagated;

FIG. 9 is a flow chart describing how changes to a subcomponent in atrack group are incorporated into the track group;

FIG. 10 is a flow chart describing how tracks in a track group arenotified of a subcomponent change;

FIG. 11 is a flow chart describing how elements of a sequence arenotified of a track change;

FIG. 12 is a flow chart describing how a component is forced to be asequence component;

FIG. 13 is a flow chart describing how a replace operation is performed;

FIG. 14 is a flow chart describing how an overwrite operation isperformed;

FIG. 15 is a flow chart describing how a range of material is replacedin a composition;

FIGS. 16A-C are flow charts describing how a subsequence is moved intoanother component;

FIG. 17 is a flow chart describing how a range of material is insertedinto a sequence;

FIG. 18 is a flow chart describing how a segment is inserted into asequence;

FIG. 19 is a flow chart describing how an insert operation is performed;

FIGS. 20A-C; are flow charts describing how a subsequence is copied;

FIGS. 21A-B are flow charts describing how a slot is made;

FIG. 22 is a flow chart describing how an extract operation isperformed;

FIG. 23 is a flow chart describing how a lift operation is performed;

FIG. 24 is a flow chart describing how a track is added to a trackgroup;

FIG. 25 is a flow chart describing how a track is removed from a trackgroup;

FIGS. 26A-C are flow charts describing how a component is split;

FIGS. 27A-B are flow charts describing how a component is dubbed;

FIGS. 28A-C are flow charts describing how a component is trimmed;

FIG. 29 is a flow chart describing how a transition is trimmed;

FIG. 30 is a flow chart describing the roll and slide editingoperations.

DETAILED DESCRIPTION

The following is a detailed description of an embodiment of the presentinvention. The method and apparatus described below may be used to editand represent multimedia compositions. A multimedia composition is acollection of relationships between time-varying media data,representing how the data should be synchronized and combined over time.Time-varying data may be, for example, video or audio data, but is notlimited to such data. Time-invarying data is a subcategory oftime-varying data and thus still pictures and text also may bemanipulated using this invention. The data are related by grouping theminto different types of components, the combination of which forms acomposition.

FIG. 1 is an illustration of a typical computer system 20 with which thepresent invention may be implemented. It should be understood that theinvention is not limited by the specific computer system shown anddescribed herein. Many other different machines may be used to implementthe invention.

Such a suitable computer system 20 includes a processing unit 22 whichperforms a variety of functions, in a manner well known in the art, inresponse to instructions provided from an application program. Theprocessing unit functions according to a program known as the operatingsystem, of which many types are known in the art. The steps of anapplication program are typically provided in random access memory (RAM)24 in machine-readable form. Because RAM 24 is typically a volatilememory, application programs are typically stored on a non-volatilememory 26, such as a hard disk or floppy disk. When a user selects anapplication program, it is loaded from the hard disk 26 to the RAM 24,and the processing unit 22 proceeds through the sequence of instructionsof the application program.

The computer system 20 also includes a user input/output (I/O) interface28. The user interface typically includes a display apparatus (notshown), such as a cathode-ray-tube (CRT) display and an input device(not shown), such as a keyboard or mouse. A variety of other known inputand output devices may also be used, such as speech generation andrecognition units, audio output devices, etc. For the purposes of theinvention, the inventors currently use a CRT display with 640 by 480pixel resolution, a keyboard and a mouse. An audio output device is alsoprovided.

The computer system 20 also includes a video and audio data I/Osubsystem 30. Such a subsystem is well known in the art and the presentinvention is not limited to the specific subsystem described herein. Theaudio portion of subsystem 30 includes an analog-to-digital (A/D)converter (not shown) which receives analog audio information andconverts it to digital information. The digital information may becompressed using known compression systems, for storage on a hard disk26 enabling it to be used at another time. A typical video portion ofsubsystem 30 includes a video image compressor/decompressor (not shown)of which many are known in the art. Such compressors/decompressorsconvert analog video information into compressed digital information.The compressed digital video information may be stored on hard disk 26for use at a later time. An example of such a compressor/decompressor isknown as JPEG III which is described in detail in U.S. patentapplication entitled "Improved Media Composer" filed Apr. 10, 1992 nowU.S. Pat. No. 5,355,450.

The computer system 20, as shown in FIG. 1, may be implemented using anumber of available commercial computer systems. The present inventionis not limited by the specific systems or computer languages shown anddescribed. At the present time, the inventors use a Macintosh IIfx,available from Apple Computer, of Cupertino, Calif., and an Indigocomputer manufactured by Silicon Graphics, Inc. of Mountain View, Calif.The computer system 20 may be programmed using any of many knowncomputer languages, in a manner with which those skilled in the art arefamiliar, to obtain and apparatus and to practice the method describedherein. The computer programming language used on the Indigo was C++; onthe Macintosh ThinkC 5.0 available from Symantec, Corp. may be used.Other suitable languages are preferably object-oriented languages. Thesystem may also be implemented in conjunction with an object-orienteddatabase system.

Using such a computer system 20, a user, such as a video editor, maycreate, edit and modify multimedia compositions, in a manner to bedescribed below, using a variety of media data.

Media data used in a composition may include original source materialsuch as video or audio tape, compact disk, computer generated images,etc. Currently available digital representations of original sources arereferred to herein as media data files.

Media data files contain digital data samples which correspond tooriginal source material, as well as information regarding how the mediadata was created, and an identification of the corresponding originalsource material. Thus, a media data file may contain a source contentidentifier, identifying the original source material. A media data filemay also include its sample rate, (and therefore the duration of asample), and the length of each sample in bytes. It also includes anindication of the section of the original source that it represents. Itstores the time offset from the source origin of its first sample. Theunits of this offset is the sample duration for the media data file.

In addition to the creation and playback information, a media data filemay include fields containing information regarding the content of themedia (e.g., "Girl in room"), the quality of the media (due tocompression), or other information.

Multiple representations of the same source material may also be storedif desired. This allows a composition to support the interchange ofmedia at different levels of visual or audio quality for differentpurposes. For example, one media data file might have a level of qualitywhich is suitable for output to videotape, while an alternative elementmight be useful for displaying in a small window on a computer screen.

A media data file may also be a representative frame of a video oranimation, or a simplified version of a graphic, or a "sound bite" froman audio source. In such cases, the function of such a media data fileis to allow a composition to approximate the actual source withouthaving to use too much disk space for storing the file.

Media data files for video may be obtained by compression, such as byusing JPEG III, or may be in other formats. The simplest, losslesscompressed data format for digital video is Run Length Encoded (RLE)RGBA data format. RLE has a simple to implement algorithm forreading/writing and compression/decompression.

A second suitable compressed video format is based on the JPEG FileInterchange Format (JFIF) standard. JPEG compression results in highcompression (sometimes 50:1 and higher) of the input data, with minimaldegradation in the appearance of the decompressed frame. This format isuseful because of its high compression/high quality characteristic, andbecause public-domain software implementations of JFIF readers/writersand compressor/decompressors are available.

A suitable graphics format is an RLE Pixel array, encoded as describedabove, but for a single frame. Graphics may also be provided in the TIFFformat, (a trademark of the Aldus Corporation) which is another formatbased on EA IFF, and allows for great flexibility in the specificationof graphic data.

A suitable format for audio data is AIFF (Audio Interchange FileFormat). The widely-used AIFF format is based on the EA IFF 85 standard.It specifies parameters such as sample size, number of channels (i.e.interleaving) and sample rate, and provides a wrapper for the raw audiodata. The AIFF format is not a compressed audio format and so there isno loss of data in translating to/from AIFF form. The Sound Designer IIformat is also an example of a suitable format.

For text files, such as commentary, file names, and other text, a mediadata file may encode text formation in ASCII format or other characterencoding.

In order to support the editing of compositions of such a variety ofmedia, the method and apparatus of the present invention includes datastructures for organizing and storing information concerning acomposition and operations for manipulating those data structures. Thesedata structures and operations will now be described in connection withFIGS. 2 through 30.

DATA STRUCTURES

The basic building blocks of a composition are called components. Acomposition is structured as a tree of components; it has a rootcomponent and a component may or may not have some subcomponents,depending on its type. A component is a function over time since itcontains information for producing the state of its portion of thecomposition at any time within its range. A component thus represents atime-dependant sequence of media data or sources called a media stream.

The simplest elements of a composition are source components, or"clips," representing sections of concrete media such as video on avideo tape, audio on a compact disk, or a computer-generated image on acomputer disk. Other components represent different combinations ofmedia streams which produce new media streams. Such components representthe variety of relationships created by editing operations performed,for example, by video editors and audio mixers.

Components may be categorized and implemented in several ways. It ispreferable to use a form of object-oriented programming to enforce acategorization. The above-described computer system 20 may be programmedusing an object-oriented programming language, such as C++, to providedefinitions of types of components. Such definitions express thecommonality between all components which are instances of a type, andenable the enforcement of structural rules for a composition.

In an object-oriented composition editing and representation system,there are two types of hierarchies. The first type of hierarchy isabstract, illustrating how one type of component may be a subtype ofanother type of component. Such a hierarchy is known as a classhierarchy and will be described below in further detail in connectionFIG. 2. The second type of hierarchy is a structural hierarchy, whichindicates how a composition is composed hierarchically of instances ofabstract components. The structural hierarchy of a composition will bedescribed in further detail below in connection with FIG. 3.

The data structures used for representing a composition exclude mediadata, by containing only indications of or references to the media dataand representations of the relationships between and combinations of themedia which form the composition. Thus, compositions are storedseparately from the media data to which they refer, and allow manycompositions to use the same media data without duplicating it. Withsuch a structure, a composition need not be reproduced when it ischanged. Further, the composition itself does not actually produce thepresentation but merely represents it and provides information for thepresentation.

In a preferred embodiment of the invention, there are several classes ofcomponents in a composition as reflected in FIG. 2. A class is acategory of object such that all objects within that class are similarin representation and functionality. These classes may in some cases beimplemented in a computer system using an object-oriented programconstruct called a class. Some of these classes are abstract classes, ofwhich no components are direct members. Components may, however, beindirect members of an abstract class by virtue of being direct membersof a subclass of the abstract class. Because there are no direct membersof an abstract class, the purpose of defining an abstract class is toexpress the commonality of the subclasses of that class. It enablesoperations to be defined once for one abstract class rather thanmultiple times--once for each subclass. Classes which are not abstract,and therefore which may have direct members, are called concreteclasses.

Specific classes will now be described in connection with FIG. 2. Itshould be understood that other and/or more classes may be implemented,and that the invention is not limited to or by the specific classesshown.

Component

The top level, abstract, class of a composition is called a component(32 in FIG. 2), and defines the functionality common to all components.For each component created for a composition, memory locations areallocated to store and group together relevant information concerningthat component. A component which is a member of a subclass of thecomponent class inherits the characteristics of the component class. Theinformation stored as a part of each component, and to be described inmore detail below is the following:

1) Track type

2) Player function code

3) Edit rate

4) Parent

5) Subcomponent identifier

6) Edit nesting level

7) Precompute

8) Name

9) Attribute list

The track type (1) is an indication of the type of material, or mediadata, represented by the component, such as video, audio, etc. Theplayer function code (2) is used to indicate the algorithm fordisplaying the material represented by the component.

A component also includes an edit rate (3) which is a representation ofthe time units used for determining the duration of the component. Theedit rate is different from the actual sample durations stored in themedia data, and can be thought of as a "virtual" sample rate. An editrate is a number of edit units per second and is not limited to being aninteger. For example, it could be a floating point decimal. The editrate thus defines the duration of virtual samples within a component. Italso determines the smallest editable unit of media. For example, aframe based 30 fps NTSC video editor may use an edit unit of 29.97 forits video components. An audio editor for editing audio associated withthe video may use the same edit rate of 29.97. Thus, media data can besubstituted at a later time with media digitized at a different samplerate, and the composition will still be valid since there is enoughinformation maintained to perform a sample rate conversion. (Sample rateconversion capabilities are well known in the art.)

A component also contains a reference to the component which depends onit in the composition, a parent pointer (4). It also contains asubcomponent identifier (5) which identifies the component in itsparent's context. Since a composition is hierarchical, every component,except the root component, has a parent component. By maintaining parentpointers it is possible, when a composition is edited, to find thedependencies on a particular component, enabling the compositionmanager, for example, readily to locate components affected by thechange.

A component may also contain an optional pointer 7, another componentrepresenting a precomputed media data file. A precomputed media datafile is one which contains a concrete representation of the intermediatemedia stream produced by the component it is attached to. This featureenables an application to play a composition in real time, when it wouldnot be possible to compute in real time the media effects represented bythe composition. Further, it enables an editor either to see thepre-computed result, or to re-make the effect from the original sourcematerial.

The edit nesting level (6) is used for identifying nested editing stepson a component. Its use will be described in more detail below inconnection with the description of editing operations.

The name (8) and attribute list (9) of a component are examples ofoptional information to assist in identifying the contents of acomponent, such as "girl in room".

With such information, a component need not maintain any explicitindication of its location in time in a composition. This location maybe computed by following the parent pointer links to the root of thecomposition, and, at each level, passing the subcomponent identifier tothe parent. The parent then computes the offset of the subcomponentwithin its own context and adds it to an accumulating offset. By notstoring this information explicitly, the number of components which needto be examined when an edit occurs may be limited, thus maintaining highefficiency of editing.

Segment 34 and transition 36 (FIG. 2) are two subclasses of thecomponent class which therefore inherit the information stored in acomponent.

Transition

Transitions, a concrete class of objects, are components which arelocated between two segments (defined below) in a sequence of components(a sequence will be defined in more detail below), and indicate how apresentation should transition from displaying the first segment todisplaying the second. Transitions act as `glue` between two segmentsand implicitly represent a combination of those segments. Thus,transitions are not independent. A transition further includes thefollowing information:

1) Transition code

2) Left effect length

3) Right effect length

The left effect length (2) and the right effect length (3) indicate theextent, in absolute value, of the transition effect, referenced to animaginary cut point between the two surrounding segments. The imaginarycut point is the point in time where the left adjacent segment wouldmeet the right adjacent segment if the transition component were notpresent. The left effect length indicates at what time the transitionstarts relative to the cut point. The right effect length indicates atwhat time the transition effect ends relative to the cut point. Theduration of a transition is determined by the sum of its left and righteffect lengths.

A transition also has a transition code (1) indicating the type oftransition to be invoked when playing the composition. Transition typesand their corresponding algorithms are well known in the art and theinvention is not limited to or by the types mentioned herein.

A transition may be as simple as a quick cut from one Segment to thenext (where the left and right effect lengths are zero) or somethingmore complicated such as a "dissolve", or "wipe" from the incomingsegment to the outgoing segment. A transition, as may any othercomponent, may have a precomputed representation of the transitioneffect. Other than this precompute attribute, inherited from thecomponent class, transitions normally do not refer to media. They may bethought of as a function of segments which precede and follow them in asequence.

Segment

A segment is an abstract class of component which represents anindependent description of a section of a presentation. It isindependent because it provides a meaningful representation of a pieceof a composition even when viewed out of the context of the rest of thecomposition in contrast to transitions which depend on neighboringcomponents. A segment may be a simple clip (representing some kind ofmedia), or it may be a more complex structure such as a sequence, ortrack group (all to be defined in more detail below). It is also therepresentation for a track. A segment inherits its functionality fromthe component class. As an example of inheritance, a segment may alsocontain a precompute attribute, which if supplied, provides access to acomputed and stored representation of that segment.

There are three subclasses of the segment class: clip 35, sequence 37and track group 38.

Sequence

A sequence, a concrete subclass of a segment, represents theserialization or concatenation in time of some other subcomponents. Asequence, as its name implies, is an ordered list of segments separatedby transitions, with the restriction that it begins and ends with asegment. This restriction allows sequences to be treated as segments,which may be included in other sequences.

The order of segments in a sequence defines their order ofinterpretation (or, "playback"). Thus, sequencing information isrepresented implicitly by the structure of the sequence, rather thanexplicitly through the use of relative start times and durations forclips. It may include the following information:

1) ordered collection of subcomponents

a) length of subcomponent in edit units

b) subcomponent identifier (pointer)

A sequence of media data is represented in the form of a tree of itssubcomponents rather than linearly. Preferably, a modification of abalanced binary tree data structure (a modified 2-3-tree) is used.Although the binary tree has been in use for some time, its applicationand modification to represent sequences of media is novel. The 2-3-treeis a tree in which each node may have either two or three subnodes. The2-3-tree is kept balanced using well-known procedures. The modificationused in the present invention relates to how a subcomponent of thesequence is found using the tree. Each node is assigned a value, or key,indicative of the total duration of any subcomponents. The keys are keptas part of the data structure of the sequence and not its subcomponent.When a position in a sequence is searched for, the keys are examined tolimit the extent of the search. Insertion procedures commonly used inb-trees are not used here, as the insertion of a segment within asequence is arbitrary, based on an editor's commands.

Clip

A clip is an abstract class of component that represents a singlecontiguous section of a piece of media, with a specified position in themedia and a specified length. A clip further contains information on itslength, which explicitly determines its duration. The position of a clipin the target media may be explicit or implicit depending on itssubclass.

There are three concrete subclasses of clips: source clips 39, trackreferences 40, and user attributes 41.

Source Clip

Compositions can represent arbitrarily complex manipulations of media,but ultimately they are all based on physical sources, such as a videotape from a camera, or a graphic produced by a computer program.References to source material are embodied in source clips 39 whichdescribe a single time-contiguous section of a source, by denoting aposition in the source, and a length.

A source clip does not contain the actual media but only references it.Source clips represent the lowest level, or leaf components, of acomposition. A source clip inherits the properties and functions of aclip, but also contains the following information:

1) Physical rate

2) Physical start

3) Physical length

4) Source identifier

5) Source offset

6) Currently linked media file

The source identifier (4) identifies a specific portion of an originalsource. A source offset (5) identifies a starting position within thesource. The selected media digitization of that source or the currentlylinked media data file (6) is represented, for example, by a pointer orfile name.

To be independent of data type and sample rate, source references suchas the source offset are measured in abstract units called edit units(EUs). An EU is a unit of duration, representing the smallest intervalof time which is recognized by a given part of a composition, and thenumber of EUs per second is called the edit rate. The edit rate for aparticular component (as mentioned above) is chosen by an editor torepresent something useful in the application domain. For example, anapplication which edits audio based on video frame boundaries would mostlikely choose an edit rate of 29.97 for the audio tracks, whereas anapplication which creates audio for use on compact discs would mostlikely use an edit rate of 44100. Another way of thinking of edit rateis as a `virtual` sample rate. This rate may or may not match the actualsample rate of the digital media.

The length of a source clip is measured in edit units, and the targetposition in the source is denoted in edit units from a point on thesource called the source origin. The origin is an arbitrary place in thesource which serves as a reference point for many measurements.

Using independent edit units for a composition may be modified to referto newly acquired media which represents the same original sources. Forexample, an editor may use highly compressed, low quality video data toperform video editing, but then use high-quality video data to assemblethe final multi-media composition. The composition can be automaticallylinked to the high-quality media data with little effort.

Further, a source clip uses position in the original source (sourceoffset) as a reference, not the position in the currently accessiblemedia data. Thus, new media data which represents a different subset ofthe source material may be linked to the composition sometime afterediting, and the composition has enough information to relocate itselfwithin the new media data. When the media data is accessed from themedia data file, the start, length and edit rate of each source clip maybe used to corresponding samples from the media data file by convertingEU's to sample durations, subtracting the offset of the media data fromits source origin, and converting the resulting sample time offset to asample byte offset.

Information concerning the media data file may optionally be stored inthe source clip as well. This information includes the sample rate,number of samples and start sample number corresponding to the desiredsource material. This material is useful for keeping a record of themedia data file used, in case that file becomes deleted and retrieval ofa similar data file is desired.

Some source clips do not refer to physical media data; the data iscomputed on the fly. These are referred to as virtual sources, orcomputed sources. These clips may be used to represent media informationthat is derived functionally, or that simulates some behavior. Much ofthe control data of a composition, such as volume settings andequalization information for audio may be represented in this fashion.For example, the control data used to "ramp up" an audio signal over a30 frame period could be represented by a virtual source.

There are special types or subclasses of source clips which representsome special cases of media: uniform source clips 42 and approximatesource clips 43.

A uniform source clip may be used to represent media that is invariantover time, such as silent audio, a "Please Stand By" graphic, or blackfiller video frames, that is to be displayed for a period of time. Byspecifying such media as uniform source clips, some special editingbehavior may be applied to those sources. For example, transitionsbetween two uniform sources may be automatically removed and replacewith one clip equal in length to the sum of the two original clips.

An approximate source clip represents media whose time base is onlyapproximate with respect to the other media in a composition. An exampleof an approximate source would be scripting information that is to bedisplayed along with a video/audio scene. The time in the scene is notproportional to the position in the script, but merely are roughlycorrelated. The text may be displayed a page at a time at theapproximate time that the actors are performing the scene, although thetiming may not be exact.

Track Reference

A track reference is a concrete subclass of a clip. It represents acontiguous section of media extracted from another track somewherewithin the same composition. The target offset into the other track iscomputed relative to the position of the track reference component inthe composition. The specific track referenced by a track reference isdefined by a track number and a scope number.

The scope number identifies a particular enclosing scope (as defined bytrack group components, described below), relative to the trackreference itself. A specific scope is found by traveling up thecomposition structure towards the root using the parent and subcomponentidentifier links, and counting each scope encountered. When the countedscopes equals the scope number, the reference scope has been located.

The track number identifies a specific track within that scope accordingto the subcomponent list of the track group.

Other implementations may identify scopes differently, for example, bysearching for a specific labelled scope, labelled with a number orstring. Other implementations may also identify tracks differently; forexample, by searching for a specific label, or by using a relative tracknumber, which is added to the referencing tracks own track number withinthe scope.

As will be described below, a track group defines a scope, and definestracks, and a track reference identifies a referenced track byidentifying a scope and a track within the scope. The track referenceobject creates a kind of relative link, which has a very usefulproperty: an assembly of components containing track references may becopied or removed from an enclosing component, and then reinsertedsomewhere else, and still by definition refer to a track in the newcontext. This feature provides the ability to interchange parts ofcompositions with ease.

Although effects may be built out of wholly contained subtracks ofinformation, it is frequently useful to create an effect whichreferences a separate parallel track in a composition. One example mightbe an audio track which contains an effect which is the echo of anotheraudio track. A track reference may be used in the effect instead of anexplicit subtrack, to make the effect on the first track refer to thesecond track. A track reference may be also used more generally forimplementing layered effects in a composition. Cut and paste of effectsis possible: a portion of a composition may be extracted and inserted ata different place in the composition or in a completely differentcomposition, while retaining the internal structural relationships ofthe contained elements.

User attributes

An arbitrary user-defined property which varies over time may also berepresented as a segment in a composition. Certain user-definedproperties may vary in time, in a manner similar to a media source suchas video. A track may be added to a composition such that it representssuch a property, and user attribute components can be added to thattrack, each representing a different value for that attribute. Forexample, a property track might be added which represents the propertycalled "Camera Distance". Components may be added to the trackrepresenting "Long Shot", "Medium Shot", or "Close up". Another propertytrack, for example, could be called "Esther's scenes", and containcomponents with the values "True" or "False", the duration of each"True" component indicating a range in the composition which containedthe actress named "Ester".

Transitions may be extended to work with attributes. A camera zooming infrom a long shot to a close up could be represented by a transitioncalled an Attribute dissolve placed between a long shot clip and a closeup clip, or an attribute dissolve between true and false in the "Ester'sscenes" track could match a visual effect of the actress fading out ofthe picture. In the context of editing the composition this transitionwould behave analogously to other transition components such as videodissolves.

Track Group

A track group 38 is an abstract subclass of a segment, which representway of combining one or more concurrent streams of media under afunction. That is, if there is to be sequencing among a number of tracksof the same or of different base media type, then those tracks aregrouped in a track group. Tracks are composed of a single segment, or asequence of segments separated by transitions. They represent a parallelrelationship of subcomponents, as compared to sequences which representa serial relationship of subcomponents. Tracks within a track group aredefined to start at the same time. Along with the information stored byvirtue of being segments, a track group includes the followinginformation:

1) Track information array

a) Mark-In

b) Mark-Out

c) Pointer to track

2) Ganged?

3) Gang length

A track group contains a list (1) of its subcomponents, or tracks. Foreach track within a track group there is a track number implied by itsposition in the list of tracks, e.g., for n tracks, a number from 0 ton-1. Other implementations may use other ways of identifying the tracks,such as by a numeric or string label attached to the track. Track groupsalso contain Mark-In and Mark-Out positions (a, b) for each track whichare used to mark places where an edit should occur. Some of the editingoperations (described below) on track groups refer to these marks whendetermining where to make a change. Also, for each track in the trackgroup, a pointer (c) is provided to indicate the component which makesup the track.

Track groups not only represent concurrency, but also define a context,called a scope, in which individual tracks may be identified. Thisfeature is used by the track reference component described above.

Track groups may also be used to represent media sources that arerelated in some fashion. For example, a track group of video data maycontain 4 sub-tracks representing 4 multi-camera angle shots. Within thetop level track, an indication could be provided to identify whichsub-track is to be played. With such a track group, a composition may becreated, and later modified on a different system, with all of theinformation that was available to the original editor.

Another type of track group could be defined to represent that a graphicon one track is to be keyed over some video on another track within thetrack group. The track group would contain information about the keycolor.

Compound audio effects may also be represented in a similar fashion, byhaving an audio track and some source clips for pan, volume andequalization settings. Such a track group would specify the way in whichthese values are to be combined to produce a single, playable track.

A special kind of concrete subclass track of group is called asimultaneous group 44 (FIG. 2). This group uses the ganged? (2) andganglength (3) fields of the track group class. For a simultaneousgroup, ganged? (2) is a boolean value set to true. This indicates thatall tracks in the group also have the same length as defined by theganglength (3).

Composition

A composition 45 (FIG. 2) is a concrete subclass of the simultaneoustrack group class, and contains one or more tracks, in which each trackmay be considered a logical player channel, and therefore is of a singlemedia type (video, audio, etc.). All tracks in a composition start atthe same point in time as defined by the track group class.

Compositions, due to inheritance from the simultaneous track groupclass, impose that all tracks within a composition are playedsimultaneously and are synchronized by the start of the composition. Inaddition, tracks within the composition are ganged (that is, they havethe same length). If a track in a composition becomes shorter duringediting than the other tracks of a composition, it is padded out withappropriated media (i.e. black video, or silent audio) to make up theshortage.

A composition created using these objects may have an arbitrarily deephierarchical structure. An example of the resulting structure is shownin FIG. 3. A composition 46 is a track group of a number of tracks 47.In this example, there are three. Each track 47 is represented as asequence in this figure. A sequence 47 is composed of a number ofsegments and transitions 48 and 49. Each segment 48 in this sequence mayalso be composed of any number of components, such as another trackgroup, another sequence, or simply a source clip. It should now bereadily apparent that such a combination represents many differentpossible types of layering of media in a composition.

OPERATIONS

The operations on the aforementioned data structures will now bedescribed in connection with FIGS. 4-30. Operations on data structuresare expressed as editing primitives, and further editing operations.Editing operations known as primitives are "split", "dub", and "trim".Split splits an existing component at a specified point into componentsof the same type, representing the media stream on either side of thesplit point. Dub produces a copy of at least part of a component, givingboth start and end points within the original component. Trim adjuststhe leading edge or the trailing edge of a component to make it longeror shorter by some number of units.

FIG. 4 shows a typical user interface 50 as it would be displayed on theoutput display device of computer system 20 and as would be seen by aneditor using this system. A composition is represented by display areas52 or 54 which are labelled "target 2" and "target 1", respectively. Adisplay region 56, labelled "source", is used to represent sourcecompositions from which material may be copied and placed into thecompositions represented by display areas 52 and 54. A display area fora composition includes Mark-In markers 58 and Mark-Out markers 60. Thesemarkers may be moved by the editor to the left and to the right toselect the portion of a composition in one of the display fields 52, 54or 56. For a composition, a representation of each track, such asdisplay regions 62 and 64 is provided. The position of the Mark-In orMark-Out marker 58 or 60 indicates a position in time on the tracks. Onepair of markers may be provided for a composition or separately for eachtrack. A track may be selected by a user by positioning a cursor, suchas a mouse cursor, on its corresponding track selector region, such astrack selector regions 66 and 68 for tracks 62 and 64, respectively.

The display 50 also includes fields representing a number of functionswhich an editor may wish to perform on a composition. Buttons 70 and 71represent overwrite and insert, respectively. Buttons 72-79 representthe functions of extract, delete, slide, roll, add, lift, create andsave, which will be described in further detail below.

It should be understood that the display interface described above andshown in FIG. 4 is merely exemplary, as many different interfaces couldbe used in connection with this invention. It is also possible toinclude a video display region for displaying the contents of a specificframe of any video track within a composition. It should also beunderstood that a number of other editing functions may be provided andthat the invention is not limited to those shown in this figure. Forexample, a variety of transitions between sequences, as described above,may be implemented and provisions for creating these transitions made inthe user interface. From the teachings in this application, theimplementation of such an interface is well within the scope of one ofordinary skill in this art.

The editing operations performed on a composition will now be describedin further detail. In general, there are two types of operations: trackoperations and component operations. Track operations are thoseoperations which are performed on portions of one or more tracks.Component operations are typically performed at the component levelwithin a composition. Typical track operations include overwriting andinserting information on one or more tracks from source tracks, andextracting or lifting material from one or more tracks.

Typical editing operations will now be described briefly in terms oftheir functionality from the editor's point of view. It should beunderstood that a number of other editing functions may be used. Aneditor may create a composition, which is, quite simply, theinitialization of a new composition component. Such initialization maybe performed either by creating an empty composition to which materialmay be added, or by selecting source material to be the basis of a newcomposition. When a composition has been created, new tracks may beadded or deleted. Within one or more tracks within a composition, aneditor may extract or lift material. Extraction of material means itsdeletion from the track wherein material following the deleted materialis concatenated with the material which preceded the deleted material.Lifting material means removing material from a track, and replacing itwith filler. Material may also be inserted into a track from a sourcetrack. This is similar to cutting the original track and placing thesource material within the cut. A portion of the track may also beoverwritten with source material, wherein the source material is used toreplace a selected portion of the target track. This is analogous torecording over a portion of a track.

Two other editing functions involve transitions at a cut between twosegments on a track. The first of these functions is called "slide" andthe second is called "roll". Slide and roll are defined in terms oftypes of trimming of transitions. One type of trim is called asymmetrical trim. With this kind of trim, the cut position is moved, butthe frame positions in the segments on both the left and right sides ofthe cuts stay fixed in time. Thus, when material is deleted from oneside of a cut, material is added to the other side from previous framesin that segment. Such frames are typically recorded on video tape or onthe digital recording media storing the source material. A second typeof trim of a transition is a left side trim. In a left side trim,material is added or deleted to the left of the cut, i.e. prior to thecut in a time reference frame. As a result, the cut is moved in time. Aright side trim involves adding or deleting material to the right, orafter, the cut. As a result, the cut is not moved in time, but materialafter the cut is shifted in time. Slide and roll are defined in terms ofleft and right side trims. A slide involves one segment located betweentwo cuts, i.e., it is in between two other segments. When sliding asegment, the transition from the preceding segment is trimmed using aleft side trim. The transition to the following segment is trimmed usinga right side trim. The effect of a slide is that the position of thepreceding and following segments remains the same, however the cutsbetween the adjacent segments and the slid segment are moved.

A roll, in contrast to a slide, maintains the transitions in the sameplace, but the source material used for the rolled segment is changed.This function is similar to scrolling through the source material. Aroll is implemented by doing a right side trim on the precedingtransition and a left side trim on the following transition.

Flow charts describing track operations, such as overwrite, insert,replace, extract and lift will now be described in further detail inconnection with FIGS. 5-28.

FIG. 5 represents a flow chart for performing a replace, overwrite orinsert operation, or other operation using both source tracks and targettracks. As the first step 80 of this process, the user selects targettracks in a manner as described above in connection with the userinterface of FIG. 4. The user may then, in a similar manner, selectsource tracks (step 82), and a mapping of the source tracks to thetarget tracks. Such a mapping may be provided by a number of methods,such as by positioning a mouse cursor on a source track and dragging thesource track to the target track. Further, "buttons" representingpossible mappings may be provided on the user interface and may beselected by the user by positioning a mouse cursor on the button. Aftersource and target tracks are selected, along with their mapping, a tracknumber list is created in step 84 which contains the mapping. The tracknumber list is, for example, an array with a length equal to the numberof tracks in the target composition. An element of the array is indexedby the track number of the target composition. The contents of anelement of the track number list array is the number of the source trackwhich is mapped to the corresponding target track. If a target trackdoes not have a source track mapped to it, the value stored in the tracknumber list is a value indicator that it is not selected.

The user also selects, for each selected track in a target and source,start and end positions (mark-in and mark-out positions). The userperforms this operation by, for example, positioning a mouse cursor onone of the markers 58 or 60 (as shown in the user interface of FIG. 4)and dragging the object to a desired location. The resulting mark-in andmark-out positions are then stored, for each track in the track group instep 88.

It should be understood that steps 80 through 84 are independent ofsteps 86 through 88 and that both of these sets of steps may beperformed in any order and any number of times by a user. However, auser typically selects tracks first, and then provide the mark-in andmark-out positions. After tracks and mark-in and mark-out positions areselected, the user selects an operation to be performed in step 90. Whenthe operation is selected, the computer performs a function, to bedescribed in more detail below, called "begin edit" (step 92). Afterthat function is performed, the operation continues with step 94 ofsetting a variable i to 0, for the purpose of performing a loop asidentified by steps 94 through 104. The effect that this loop is thatfor each target track which is selected, the user selected operation isperformed using that track and the source track which is mapped to it.In order to do this, the with entry of the track number list is examinedin step 96. If that entry does not contain a track number value, thevalue i is incremented by one in step 98. It is then determined in step100 whether the current value of i is greater than the length of thetrack number list. If i is still an element of the track number list,the processing continues with step 96 as described above.

If, in step 96, it is determined that the with entry of the track numberlist contains a track number, a value j is set to be that track numbervalue in step 102. The selected portion of track i is then forced to bea sequence in a manner to be described below in connection with FIG. 12.The selected operation is then performed on track i using source trackj, in step 104. The details of the processes of the selected operationswill be described in more detail below in connection with FIGS. 13through 30. Upon completion of the operation, processing continues withsteps 98 and 100 until all of the elements of the track number list havebeen examined. At this point, the edit is complete and an editdescriptor is created in step 106, the process of which will bedescribed in more detail below. The operation is completed by performinga function, hereinafter called "end edit", which will be described inmore detail below in connection with FIG. 7.

A flow chart for track operations which do not use a source track willnow be described in connection with FIG. 6. The first few steps forthese types of track operations, such as extract and lift, are similarto those for track operations which use source tracks. Thus, the userselects target tracks in step 80 and a track number list is createdcontaining a mapping in step 110. The mapping is different in this casein that a track number list merely has an indication of whether or notit has been selected, and any valid track number may be used. The useralso selects-mark-in and mark-out positions for each selected track insteps 86 through 88 which are identical to steps 86 and 88 of FIG. 5.Again, steps 80 and 110 may be performed independently as steps 86 and88. After the user has selected a target track and the mark-in andmark-out positions, an operation is selected in step 112, such asextract or lift. The computer then processes the operation by performingthe "begin edit" function in step 114. The computer steps through eachentry of the track number list to determine whether the correspondingtrack had been selected, forces the selected portion of the track to bea sequence, and performs the operation on each selected track. Thecomputer performs these steps using steps 116 through 124 which aresimilar to steps 94 through 104 as described above in connection withFIG. 5. The specific operations performed in step 124, such as extractand lift, will be described in more detail below in connection withFIGS. 22 and 23. When the operation has been performed on each track, anedit descriptor is created in step 126 and the end edit function isperformed in step 128, in a manner similar to steps 106 and 108 of FIG.5.

The "begin edit" function, as mentioned above, will now be described.This function operates on a component by incrementing the edit nestingcount for that component. It establishes the beginning of an editoperation which is composed of multiple editing steps, and is used laterto prevent propagation of an edit description for each step. For theoperations described in FIGS. 5 and 6, a begin edit is performed on thetrack group and increments the edit nesting count for that track group.For example, if the track group is a composition, the edit nesting countof the root component is incremented.

The creation of an edit descriptor (step 106 of FIG. 5 and step 126 ofFIG. 6) will now be described. The edit descriptor is a data structurethat captures significant features of a change to a component due to anedit. It contains the left most time affected the change (the leftbound), the right most time affected by the change (the right bound).The right bound may be a special value indicating that the change is notbounded on the right. The value is preferably the largest integer.Within the changed region, part of the region may be designated as atime shifted region. The offset of the shifted region from the leftbound is stored as the sync offset. The amount and direction of theshift in the shifted region is also part of the edit descriptor, andwill be called hereinafter as "sync shift". The edit descriptor could beexpanded to include other parameters describing more detail about thechange.

The edit descriptor generated for replace, insert, overwrite, extractand lift commands will now be described. For all of these operations,the left bound is the mark-in position on the target track. For replaceand overwrite, the right bound is the mark-out position on the targettrack. For both of these functions, the sync offset and sync shift areboth 0. A replace is similar to an overwrite, except that if thereplaced portion has a length which is different from the portionreplacing it, the right bound is unbounded. The difference between thelength of the source material and the length of the target material isthe sync shift which begins at a sync offset defined by the minimum ofthe length of the source material and the length of the target materialreplaced. The insert and extract functions both have a right bound whichis unbound, and an offset of 0. The sync shift for an insert is thelength of the inserted material. Similarly, the sync shift for anextract is the negative of the length material extracted. Editdescriptors for other types of functions will be described in moredetail below as the functions are described.

The "end edit" function will now be described in connection with flowchart of FIG. 7. This function establishes the end of the edit operationand propagates the corresponding edit description, unless the operationwas nested inside another.

This operation is performed on a particular component and an editdescription. For example, in steps 108 (FIG. 5) and 128 (FIG. 6) the endedit function is performed on the target track group component. Thefirst step 130 of the end edit function is decrementing the edit nestingcount of the component on which it is operating. If the edit nest countbecomes 0, as determined in step 132, and if the component is the rootcomponent of the composition, as determined in step 134, this operationis complete. If the edit nest count is not 0, this function isterminated. If the component was determined, in step 134, not to be theroot of the composition, the edit description is propagated to theparent component in step 136. This propagation will be described in moredetail in connection below with FIG. 8.

The propagation of an edit description to a component, from its childcomponent will now be described in connection with FIG. 8. If the editnest count of the component is equal to 0, the change, described by theedit descriptor, to the subcomponent is incorporated into othersubcomponents of this component in step 140. This incorporation involvesperforming any necessary operations on a component to react to thechange in its subcomponent. This operation will be described forparticular components in more detail below.

After step 140 is completed, if the component is the root component, asdetermined in step 142, this process ends. Similarly, if the edit nestcount, as determined in step 138, was equal to 0, processing also ends.If the component is not the root component, a new edit descriptor iscreated which describes changes to this component after the subcomponentchange was incorporated (step 144). That is, the edit description forthe subcomponent is translated into the time reference frame of this,its parent, component. For a sequence, the left bound and right boundare simply increased by the offset of the subcomponent within theparent. For a track group, the edit descriptor remains the same. Afterthe new edit descriptor is created, it is propagated to its parent andsteps 138 through 146 are repeated for that parent and any furtherparents.

The incorporation of a subcomponent change into its parent component, asmentioned in step 140 of FIG. 8, will now be described in connectionwith FIGS. 9-13. The function of this operation is specific to the kindof component which is applied. It operates on a component along with itsedit description and the identifier of the subcomponent which waschanged. The function may be different for each type of component, andperforms any necessary work in a component for reacting to a change inone of its subcomponents.

For track groups, a change in a subcomponent may cause an edit to occuron another track in the same track group such that track relationshipsenforced by the track group are maintained. For a sequence, this step isperformed by modifying the 2-3 tree representing the sequence to assigna new value to the subcomponent indicated by the identifier of thatsubcomponent. The modification of the 2-3 tree is based on the valuesprovided from the edit description lists. The change in the subcomponentis propagated up to all higher parent nodes in the tree.

The incorporation of a subcomponent change for a track group componentwill now be described in connection with the flow chart of FIG. 9. Thisprocedure begins with the "begin edit" function as applied to the trackgroup component (step 162). If the track group is ganged, as determinedin step 164, it is then determined whether the new subcomponent islonger than the given gang length (step 166). If the subcomponent is notlonger than the gang length, filler is added to the end of all shortertracks in the track group to make them equal to the new gang length(step 168). Otherwise, it is determined whether the subcomponent isshorter, in step 170, and if it is, a filler is added to the changedtrack to bring it to the correct gang length in step 172.

If the track group is not ganged, or after a gang track group has beenmodified so that all tracks are of the same length, for each track inthe track group, except the changed track, a notification is sent instep 175 to that track, to propagate change information downwards belowthe changed subcomponent of the composition. This notification step maybe implemented differently for different types of components sincedifferent components have different constructions.

Notification operates on a component and includes a scope depth, arelative track number and an edit descriptor as its parameters. Thescope depth is used to measure how many track groups have been traversedin a downward propagation. The relative track numbers used to indicatethe distance between the track that changed and the track that is beingnotified. The scope depth is originally set to zero for the track groupand which a subcomponent change was originally incorporated. Therelative track number is the difference between the track number of thechanged subcomponent and the current track number. The edit descriptoris that for the track group containing all the tracks to be notified.

The notification to a track which is in itself a track group will now bedescribed in more detail in connection with FIG. 10. The first step ofthis process is incrementing the scope depth as shown in step 176. Thenotification procedure is then recursively applied to each of the trackswithin the notified track group in step 178. This function has theeffect of passing down the notification of a change to all subtrackspreserving the relative track number and edit description, butincrementing the scope depth.

The notification of a track change as applied to a sequence will now bedescribed in connection with FIG. 11. The first step is step 180 offinding the component in the sequence which is to the left of thechanged subcomponent. This step is used by searching the sequence 2-3tree for the component at the value indicated by the left bound value ofthe edit descriptor. Similarly, in step 182, the component to the rightin the sequence is also found, using the right bound value of the editdescriptor. For each component between the left component and rightcomponent, inclusive, a new edit descriptor is created from the previousedit descriptor, adjusting it for the offset of the subcomponent withinthe sequence (step 184). Further, for each subcomponent the notificationof the track change is propagated, according to the subcomponent type(step 186). The effect of this procedure is that the notification ispassed down to all subcomponents of a sequence which are within the timebounds of the change.

FIG. 12 is a flow chart describing how a section of a track is forced tobe a sequence, as mentioned above in connection above with the flowcharts of FIGS. 5 and 6 (steps 103 and 123). The flow chart of FIG. 12describes a function which operates on a track group and a track number.If the specified track of a track group is not a sequence component,this function turns it into a sequence. This function is used becausethe track editing operations of REPLACE, OVERWRITE, INSERT, EXTRACT andLIFT act on sequences. The first step of this process is step 150 offinding the component indicated by the marking position in the specifiedtrack (step 150). If the found component is a sequence, as determined instep 152, processing is complete. Otherwise, an empty sequence componentis created in step 154. The original component is removed from the trackgroup in step 156 and replaced, in step 158 with the empty sequencecomponent. The original component is inserted into the sequence in step60. Then this process is complete.

The procedures for performing the replace, overwrite and insertoperations will now be described in connection with FIGS. 13 through 28.FIG. 13 describes the replace operation. The first step is settingvariables left offset, right offset, source left offset and source rightoffset to values to be used for the replace operation (step 190). Theleft offset is set to the mark-in position on the target-track and theright offset is set to the mark-out position on the same target track.The source left offset and source right offset are respectively set tothe source track mark-in position and the source track mark-outposition. With these variables set, the indicated range in the targettrack is replaced with the indicated source track material in a mannerto be described in more detail below (step 192) in connection with FIG.15.

FIG. 14 describes an overwrite operation, which is similar to a replaceoperation. The first step 194 involves setting the left offset to thetarget track mark-in position and the right offset to the sum of thetarget track mark-in position and the difference between the sourcetrack mark-in and mark-out positions. The source left offset and sourceright offset are respectively set to the source mark-in and mark-outpositions. (Step 194). With these variables set, the indicated range inthe target track is replaced with the material indicated from the sourcetrack in step 196 in a manner to be described below in more detail inconnection with FIG. 15. This step is identical to the step 192 of FIG.13 describing a replace operation.

Replacing a range in a target with a source component will now bedescribed in connection with the flow chart of FIG. 15. This procedureoperates on a source component, the track containing the selected sourcematerial, and a target sequence. The target sequence may have been madeby the force-to-sequence operation described above in connection withFIGS. 5, 6 and 12. The first step 200 of this procedure is performingthe "begin edit" function on the target sequence (step 200). The rangeof the target sequence, as determined by the left offset and rightoffset values, is then deleted in step 202. This step 202 involvesmoving the subsequence indicated by the left offset and right offsetvalues to a known destination, which will be described in more detailbelow in connection with FIGS. 16A-C.

After the range of the target has been deleted, the source materialdetermined by the source left offset and source right offset from thesource track is inserted into the target sequence at the positiondefined by the left offset (step 204). This process will be described inmore detail below in connection with FIG. 17. After the source materialis inserted, an edit descriptor is created in step 206 for the changedregion and the "end edit" procedure, as described above, is performed instep 208.

The process of moving a subsequence will now be described in more detailin connection with FIGS. 16A-16C. This operation, given a sourcesequence, a destination sequence, left and right offsets in the source,and a destination offset in the destination sequence, removes thecomponents between the left and right offset from the source and insertsthem at the destination offset in the destination sequence. If thedestination sequence is specified as null, the removed components aredeleted. The alternation of segments and transitions within a sequenceis preserved by replacing the removed components with a cut component inthe source and by surrounding the moved components in the destination bycuts.

The first step 240 of moving a subsequence is performing the "beginedit" function on the source sequence. If the destination sequence isnot null as determined in step 242, the begin edit function is performedon the destination sequence in step 244. After steps 242 and 244, a slotis made at the left offset in the source in step 246, in a manner to bedescribed in more detail below in connection with FIGS. 21A-B.

If the left offset is equal to the right offset, as determined in step248, there is no need to make a slot at the right offset position in thesource, otherwise a slot is made in the source at the right offset instep 250. After a slot, if any, is made at the right offset, if thedestination sequence is not null, a slot is made at the destinationoffset in the destination sequence in step 254. A cut transitioncomponent is then created and inserted in the resulting slot in step256. After the slot and cut transition are made, or if the destinationis null, for each component in the source between the slots the left andright offsets, the component is removed from the 2-3 tree in the source.If the destination is not known, the component is inserted in the slotmade in the destination sequence, following the previously insertedcomponent, if any. If the destination is null, the component is merelydeleted.

When all the components have been removed from the source, and if thedestination is not null, a cut is created and inserted in thedestination sequence following the previously inserted component in step268. Next, a cut is created and inserted in the source in the positionwhere the components were removed (step 270). If the destination is notnull, and an edit descriptor is created for the changes in thedestination sequence and the end edit function is performed on thedestination sequence. Next, an edit descriptor is created in step 276,for the changes in the source, and the "end edit" function is performedon the source in step 278.

The step of inserting a range of source material into a target sequence(as performed in step 204 of FIG. 15) will now be described in moredetail in connection with FIG. 17. This procedure makes a new component,representing part of a source component, and inserts the new componentinto the target sequence. If the source is a sequence, instead of makingand inserting a new component, which would result in a sequence nestedinside a sequence, this procedure provides that individual relevantsubcomponents of the source sequence are copied and inserted into thetarget sequence.

This procedure begins with the "begin edit" procedure as performed onthe target sequence (step 210). If the source track is a sequence, asdetermined in step 212, the subsequence, as determined by the sourceleft offset, and source right offset is copied from the source track tothe target track at the location marked by the left offset (step 214).This step of copying will be described in more detail below inconnection with FIGS. 20A-C.

If the source track is not a sequence component, the range of the sourcematerial from the source left offset to the source right offset isdubbed in step 216 to result in a new segment component. This process ofdubbing will be described in more detail below in connection with FIGS.27A-B. The dubbed segment is then inserted into the target sequence atthe position defined by left offset (step 118), in a manner to bedescribed in more detail below in connection with flow chart of FIG. 18.

After the subsequence is copied in step 214, or the segment is insertedin step 218, an edit descriptor is created in step 220 and the "endedit" function is performed on the target sequence in step 222.

The process of inserting a segment, as used in step 218 of (FIG. 17)will now be described in connection with the flow chart of FIG. 18. Thisprocedure, given a position in a sequence, inserts a new segment intothe sequence at that position. For the process of inserting a range ofsource material into a target sequence, that position is the leftoffset.

This process begins with the "begin edit" procedure of step 224, afterwhich a slot is made, in step 226, at the selected position in thetarget sequence, in a manner to be described below in connection withFIGS. 21A-B. When a slot is made, a value is returned which indicatesthe subidentifier of the component to the left of the slot made. A cuttransition, the segment, and another cut transition are then insertedinto the 2-3-tree of the sequence, immediately following the componentwith the subidentifier returned by the making of a slot (step 228). Thisstep inserting the transition segments in the tree is a standard treeoperation, as described above in connection with the description of the2-3-tree for representing sequences. After the segment is inserted instep 228, the corresponding edit descriptor is created in step 230 andthe "end edit" function is performed on the target sequence in step 232.

The editing operation for inserting material from a source track into atarget track will now be described in connection with FIG. 19. The firststep of this editing operation is setting the left offset value to themark-in position of the target track. Source left offset and sourceright offset values are respectively set to the mark-in and mark-outpositions of the source track. The range defined by the source trackmark-in and mark-out positions is then inserted into the target track atthe position defined by left offset in step 236. This step 236 ofinserting a range is described above in connection with FIG. 17.

The copying of a sequence, such as for step 214 of FIG. 17, will now bedescribed in connection with FIGS. 20A-20C. The step of copying of asequence is similar to moving a subsequence, but instead of removingsubcomponents from the source, the components are merely copied and thesource remains unchanged. Thus, when the offsets for the source fallwithin a component of the source, a partial copy is made by dubbing,rather than making a slot (which is used in moving the subsequence) inorder not to modify the source.

The first step 280 of copying a sequence is finding the component in thesource at the position of the left offset. A sub-left offset is thengenerated in step 282 by subtracting the offset of the beginning of thatcomponent within the sequence from the left offset value. Next, acomponent at the right offset position in the source is found in step284, and similarly a sub-right offset is obtained in step 286 bysubtracting the offset of the beginning of the right component withinthe sequence from the right offset value. With these components andvalues determined, the next step is performing the "begin edit" functionon the destination sequence (step 288).

If the right component and left component are the same, the leftcomponent is dubbed, to make a copy of the component between thesub-left offset values and the sub-right offset values (step 290). Thiscopy is then inserted into the destination sequence at the destinationoffset in step 292, in a manner as described above in connection withFIG. 18. Next, an edit descriptor is created to describe the change tothe destination sequence (step 294 of FIG. 20C) and the "end edit"function is then performed on the destination sequence in step 296.

If the left component and the right component are not the same, asdetermined in step 289 (FIG. 20A), a slot is made at the destinationoffset in the destination sequence (step 298) in a manner describedbelow in more detail in connection with FIGS. 21A-B. Next, a cuttransition is created and inserted in the slot in the destinationsequence in step 300. Next, the left component, starting at the positiondefined by sub-left offset, is dubbed, to create a copy in step 302.This copy of a partial component is then inserted into the destinationsequence following the previously inserted cut (step 304). Then, foreach component after the left component which is to be copied and theright component, a copy of the component is made in step 306 and thatcopy is inserted into the destination sequence following any previouslyinserted component (step 308).

If the last component inserted is the right most component of the copiedsequence, a cut transition is created and inserted into the destinationsequence following this last component in step 312. Otherwise, the rightcomponent is dubbed, starting at offset 0 and ending at the sub-rightoffset, in step 314, which copy is then inserted into the destinationsequence following the last component previously inserted into thedestination sequence in step 316. After the right component has beeninserted into the destination sequence, the cut is created in step 312as described above. When the copy is complete, the edit descriptor iscreated in step 396 and the end edit function is performed on thedestination sequence in step 296.

The process of making a slot in a target sequence, at an offset, forexample, as used in step 298 of FIG. 20A, 226 of FIG. 18, 246 of FIG.16A, will now be described in connection with FIG. 21A-C. The functionof making a slot is used to prepare a sequence or subcomponents to beinserted at a position indicated by an offset. In a properly formedsequence, subcomponents are arranged in alternately series of segmentsand transitions where the first and last components are segments. Thesegment that is located in the target sequence at the indicated offsetis found and split into two segments at the indicated offset. Thisoperation leaves the sequence in a temporarily inconsistent state inwhich two segments are adjacent to each other without a transition inbetween. The point between the segments is considered a slot. This slotis a place where a transition component, or an alternating series oftransition components and segment components bounded by transitioncomponents, can be inserted.

The slot making procedure will now be described in connection with FIGS.21A-B. The first step 320 of this process is finding the component thatfalls at or covers the position indicated by the offset where a slot isto be made. The slot sub-offset is then set, in step 322, to be thedifference between the offset and offset of the found component withinthe sequence. If the slot component is the first component, asdetermined in step 324, the transition directly to the left of thecomponent is then examined in step 326, to determine its right effectlength. If the right effect length is determined to be greater than theslot sub-offset, as in step 328, an error is reported.

Many things may be done in response to an error, because it reallyindicates an undefined request on the part of the editor. That is, it isnot known whether the transition effect length should be shortened, orif the same transition should be placed on opposite sides of the slotwhere something is to be inserted, or if the editor did in fact make amistake. For the sake of simplicity, the attempted edit may be aborted,and to revert the composition back to its original state before editing.

If the transition to the left of the component, if any, is not affectedby the slot making procedure, the process continues with step 330 ofdetermining whether the slot component is the last component of thesequence in which the slot is made. If it is not the last component, thetransition to the right of the component is examined in steps 332 and334 to determine if its left effect length is greater than the length ofthe slot component minus the slot sub-offset. As above in step 328, ifthe adjacent transition is affected by the slot, an error is reported.If the transition is not affected, or if the component is the lastcomponent in the sequence, the process continues through step 336 ofsplitting the component at the position indicated by the slot sub-offsetto break off a new component either from the left or the right of thegiven component.

If the new component is generated on the left, as determined in step 338based on information provided from the splitting of the component, thenew component is inserted to the left of the slot component in step 340,otherwise it is inserted to the right of the slot component in step 342.The insertion is performed according to the insert component proceduredescribed above in connection with FIG. 5. After the new component isinserted, the value of the original component, indicating its duration,and the tree corresponding to the sequence in which the component isfound, is shortened to reflect the shortened length. The original slotcomponent remains in the sequence, but its size is merely changedbecause a piece has been broken off by the split operation in step 336.The split operation will be described in more detail below in connectionwith FIGS. 26A-C.

The editing operation of extracting material from a target sequence willnow be described in connection with FIG. 22. The first step 400 of thisprocess is creating a new empty sequence. The subsequence indicated bythe left and right offsets, as determined by the mark-in and mark-outpositions, is then moved into the created new sequence at offset 0 (step402). This step is performed using the process described above inconnection with FIG. 16A-16C. The extracted sequence is then returned,for example, by returning a pointer to its location at memory, in step404.

The editing operation for lifting a sequence from a target sequence willnow be described in connection with FIG. 23. The first step of thisprocess is performing the "begin edit" function in step 406. Thematerial as indicated by the mark-in and mark-out positions on thetarget sequence, i.e. the left and right offsets, is extracted, usingthe process described above in connection with FIG. 22. A filler objectis then created in step 408, having as its length, the length of theremoved sequence. The filler is then inserted, in step 412, using theprocess for inserting a segment as described above in connection withFIG. 18. An edit descriptor is then created in step 414 and the end editfunction is performed in step 416. The lifted material is then returned,in step 418, for example by providing a pointer to its location inmemory.

If a delete operation is to-be implemented, i.e.; one which does notsave the extracted material, it may be implemented by modifying theextraction procedure of FIG. 22 to eliminate the creation and returnsteps 400 and 404, and by moving the extracted subsequence to a nulldestination.

FIG. 24 describes another editing function for adding a track to a trackgroup. The first step of this process is performing the "begin edit"function in step 420. The new track is initialized in the track group instep 422, by creating a new track object, and initializing its storedvalues. The track is given a user selected track number according theselected insert position in the track group. The initialized trackinformation is stored in the list of tracks for the track group, and thecomponent for the track stores its track number and an identifier to thetrack group as its parent.

If the track group is ganged, the tracks are kept the same length byadding or removing filler. The addition or removal of filler may beperformed in a manner as described above for replacing or inserting acomponent. Next, in step 426, the subidentifiers, or track numbers, ofother tracks in the track group having a track number are originallyequal to or higher than the inserted track number, or incremented. Thenother tracks other than the newly inserted tracks are notified of thisinsertion in step 428. The process for notification is the same as thenotification of a track change, as described above in connection withFIG. 11. Next, an edit descriptor is created in step 430, whichindicates the difference in length of the track. The "end edit" functionis then performed in step 432.

The removal of a track will now be described in connection with FIG. 25.This process begins with step 440 of performing the "begin edit"function on the track to be removed. Next, the track is removed from thetrack group, by deleting its information from the track group array, orlist, and by setting its parent and subidentifier values to no, orsimilar nonrepresentative values. When the track is removed in step 442,a pointer is returned to the track, enabling its use by an editor forother compositions. The subidentifiers for other tracks having highertrack numbers in the track group are then decremented in step 444. Theremaining tracks are then notified in step 446 that a track has beendeleted. This step is performed in the same manner as the notified trackchange described above in connection with FIG. 11.

If no tracks remain in the track group, the track group's gang length isthen set to 0 in step 448. The track removal process is completed thenwith the steps 450 and 452 respectively of creating an edit descriptorand performing the "end edit" function on the track.

The primitive operations of split, trim, dub and trim transition willnow be described in connection with FIGS. 26 through 29. The operationof split is shown in FIGS. 26A through 26C, as it may be performed ondifferent components.

A split is a recursive function that operates on a component. Every typeof component implements a split function. When split is applied to acomponent at a specified position, the component modifies its ownstructure into one that represents the material on one side of theindicated position, and creates a new component representing the otherside. The choice of which side (left or right of the position) is to bethe new component is made dynamically based on performanceconsiderations. That is, the shorter part of the component, whichrequires less copying, is moved to the new component.

The operation of split on a sequence component, given a specifiedposition will now be described in connection with FIG. 26A. The firststep 460 of splitting a sequence is making a slot in a sequence at theposition indicated. The sequence component is duplicated except for itstree of subcomponents in step 462 and in the new component a new emptytree is created in step 464. Next, it is determined in step 466 whetherthe number of components on the left of the slot is greater than thenumber of components on the right of the slot. For the side which hasfewer components, each component on that side is removed from theoriginal sequence and inserted into the newly created sequence,preserving the order of the subcomponents (steps 468 or 469).

Splitting a track group will now be described in connection with theflow chart of FIG. 26B. Splitting of a track group involves recursivelysplitting all of its subcomponents representing its tracks, and placingthe new components into a new track group. The first step of thisoperation is duplicating the track group component except for itspointers to subcomponents step 470. For each subcomponent attached tothe track group, the component is split into two components in step 472.The left part of the split subcomponent is placed in the original trackgroup in step 474, while the right part of the split component is put inthe new track group in step 476. When this loop has been completed foreach subcomponent, splitting of a track group is complete.

Splitting of a source clip will now be described in connection with FIG.26C. Since source clips have no subcomponents, splitting them isrelatively simple. The first step is duplicating the source clip in step480. The length of the original is set to be the position where thesplit is indicated in step 482. The length of the copy is set to be theoriginal length minus the split position in step 484. The source offsetof the copy is then set to be the original source offset plus with splitposition in step 486.

Dubbing of a segment will now be described in connection with FIGS. 27Athrough 27C. Dub is a recursive function on a component and itssubcomponents. Given a component, a left offset and a right offset,dubbing makes a new component of the same type, representing thematerial between those offsets. Dub may be implemented for allcomponents but its details may vary by component type.

Dubbing of a sequence will now be described in connection with FIG. 27A.The first step is duplicating the sequence component as indicated by theleft and right offset, except for its tree of subcomponents. A new emptytree is made in the copy in step 492. The indicated range of theoriginal is then copied, using copy subsequence, as described above, anda copy of the new component is inserted into the copy of the sequence(step 494).

Dubbing of a track group will now be described in connection with theflow chart of FIG. 27B. The first step is duplicating the track groupcomponent except for its subtracks 500. For each subtrack, thesubcomponent is dubbed in step 402 and the copy of the subcomponent isinserted into the new track group in step 504. Dubbing of a track groupis completed when dubbing of each subtrack is complete.

Dubbing of a source clip is relatively simple, as it involves merelycreating a new source clip as a duplicate of the original source clipcomponent.

The trim operation will now be described in connection with FIGS. 28Athrough 28C. Trim is an operation which modifies the edges of anycomponent. It recursively operates on the subcomponents of a component.Given a flag indicating the left edge or the right edge of a component,a value "delta" indicating the change and position of that edge, and aflag indicating whether destruction trimming is allowed, a trim isperformed. Destructive trimming is a trim which at some level of thecomposition will cause a component to be deleted.

Trimming a sequence will now be described in connection with FIG. 28A.The first step is performing the begin edit function on the sequence510. If the left edge is to be trimmed, a variable "edge component" isset to be the rightmost component of the sequence, otherwise thevariable is set to be the leftmost component of the sequence. If theamount of trim is greater than the length of the edge component, asdetermined in step 518, and if destructive trimming is not allowed, asdetermined in step 520, an error is reported. If destructive trimming isallowed, the function "delete range" is performed (steps 524 and 526),for the edge of the component to be trimmed, which is determined in step522.

If the amount of trim is less than the length of the edge component, theedge component is trimmed by the amount "delta" as mentioned above instep 530. After the edge component is trimmed in step 530, the tree ofthe sequence is modified to reflect the change value of the edgecomponent.

After a range is deleted, or a component is trimmed, this procedureterminates by creating an appropriate edit descriptor and by performingthe "end edit" function.

Trimming of a track group will now be described in connection with FIG.28B. The first step 540 of trimming a track group is performing thebegin edit function on the track group. If the track group is ganged, asdetermined in step 542, the gang length is adjusted by the amount oftrim in step 544. After the gang length is adjusted, if necessary, foreach subtrack in the track group, the subtrack is trimmed in step 546.When all of the subtracks have been trimmed, an edit descriptor iscreated in step 548 and the end edit function is performed in step 550.

Trimming of a source clip will now be described in connection with FIG.28C. The first step 552 of this operation is performing the begin editfunction. If the left edge of the source is to be trimmed, the sourceoffset is adjusted by the amount of trim in step 556. After anyadjustment to the source offset, the length of the source clip isadjusted by the amount of trim in step 558. An edit descriptor is thencreated in step 560 and the end edit function is performed in step 562.

The process of trimming a transition will now be described in connectionwith FIG. 29. This operation uses the subcomponent identifier of atransition within a sequence, a left delta indicating how much to trimthe component on the left of the transition, a right delta indicatinghow much to trim the component on the right and a flag indicatingwhether destruction of subcomponents is allowed. The first step 570 oftrimming a transition is finding the transition component from thesupplied subcomponent identifier in step 564. A variable "leftcomponent" is then set to be the component to the left of thetransition. The left transition is the transition to the left of thecomponent (568). Similarly, the right component and right transition areset in steps 570 and 572 to be the component to the right of thetransition and the transition to the right of the right component.

If the sum of the right effect length of the transition and the lefteffect length of the right transition is greater than the length of theright component, an error is reported (step 574). If the sum of the lefteffect length of the transition in the right effect length of the lefttransition is greater than the length of the left component, it isdetermined in step 576, and another error is reported.

The next step 578 of trimming a transition is performing the "beginedit" function on the transition. The right edge of the left componentis then trimmed by the left delta value in step 580. The flag indicatingwhether destruction of subcomponents is allowed is passed to this trimfunction. Next, in step 582, the left edge of the right component istrimmed by the right delta value. Finally, the trimming of transition iscompleted by steps 584 of creating the edit descriptor and 586 ofperforming the end edit function on this transition.

The procedures for performing slide and roll operations will now bedescribed in connection with FIG. 30. The initialization procedures forboth the slide and roll, wherein the user selects a target track andmark-in and mark-out position is similar to those described above inconnection with FIG. 6 and the extract and lift functions. The maindifferences with these functions is that they also take a direction andan amount in that direction. Such an indication can be provided by avariety of user device inputs, such as by dragging a mouse cursor or bydepressing a key for a fixed amount of time on a keyboard.

The mark-in and mark-out positions are actually used to determine theidentifiers of the transitions, as the transitions stay in place and arenot moved. The first step of a roll is performing the "begin edit"function in step 600. If the distance or duration of the roll is greaterthan 0 or positive, as determined in step 602, the right side transitionis trimmed first in step 604 and then the left side is trimmed in step606. This insures that the rolled sequence gets longer before it getsshorter, so the edges do not run into each other. Otherwise, if thedistance is negative, the left transition is trimmed first in step 608,then the right in step 610. After the appropriate trims are performed,the edit description is created in step 612 and the end edit function isperformed in step 614. The edit descriptor includes the left bound asthe offset of the left transition and the right bound as the offset ofthe right transition. The offset of synchronization begins at an offsetof 0 from the left bound, and the amount of the shift is the distance ofthe roll.

A slide is similar to a roll, although it uses different trimmingoperations on the left and right sides as described above. However, theorder of the trim operations for positive and negative distance valuesmay be arbitrary. The edit descriptor for a slide is also different. Theleft bound value is the minimum of the position of the transition andthe sum of this position and the distance of the slide. Similarly, theright bound is the maximum of the position of the right transition andthe sum of the right transition and the distance of the slide. Theoffset of any time shift is 0, and the amount of the time shift is thedistance of the slide.

By implementing the above-described functions, a comprehensive editingsystem using a representation in accordance with the invention may beimplemented. It should be understood that these functions may beimplemented in a variety of different ways and that more or lessfunctions may be provided, as desired or needed.

There are a variety of ways in which a composition in accordance withthis invention may further be used to present the information itcontains. It may be played back, in order to be viewed by a person, suchas its editor. An edit decision list may be created in order to controlthe original data sources to produce the final presentation or anothervideo tape. Finally each frame may be randomly accessed for editing orviewing.

Having now described an embodiment of the present invention, it shouldbe understood that the foregoing is merely illustrative, having beenpresented by way of example only. Numerous modifications may be made tothe embodiments shown, and such modifications are considered to bewithin the scope of the invention as defined by the following claims andequivalents thereto.

What is claimed is:
 1. A computer-readable medium containingcomputer-readable logic defining a data structure stored thereon forrepresenting a multimedia composition and for access by an applicationprogram being executed on a data processing system, the data structurecomprising:a first track group of a plurality of tracks of themultimedia composition, wherein each track is defined as a sequence ofcomponents sequentially related in time, wherein each component isassociated with content defined by one or more temporal samples of mediadata and wherein all tracks are displayed concurrently beginning at acommon time; and wherein content of a selected component of the sequenceof components defining one track of the plurality of tracks in the firsttrack group is defined by a track reference, wherein the track referencehas a time value with reference to the common time and an indication ofanother track of the plurality of tracks in the track group and aduration starting from the specified time, wherein the specified timeand the duration are used by the application program to identify one ormore components in the other track, wherein the content associated withthe identified components is used to define the content of the selectedcomponent.
 2. The computer-readable medium of claim 1, wherein acomponent in the sequence of components in one of the plurality oftracks includes a second track group comprising a plurality ofconcurrent tracks, each track having components sequentially related intime, and wherein content of one of the components in the second trackgroup includes the track reference, wherein the track reference includesa scope number to identify which of the first and second groups oftracks is selected.
 3. The computer-readable medium of claim 1, whereinone component in the sequence of components defining one track in theplurality of tracks defines an effect, wherein the effect is representedby an indication of the effect to be performed and wherein the contenton which the effect is performed is defined by the track reference. 4.The computer-readable medium of claim 1, wherein each track in a trackgroup has a track number, and wherein the track reference defines theother track using the track number of the other track.
 5. In combinationwith the computer-readable medium of claim 4, means for displaying themultimedia composition using the representation, wherein the means fordisplaying includes means for displaying the component defined by thetrack reference by identifying a track group using the scope number, byidentifying a track within the identified track group using the tracknumber, and by identifying a component in the identified track to accessthe content defined for the identified component.
 6. An object-orientedsystem for representing a multimedia program for implementation on acomputer in an object-oriented framework; comprising:(a) an abstractcomponent class; (b) a track group class derived from the abstractcomponent class and defining a first subclass of the abstract componentclass, wherein a track group object, an instance of the track groupclass, comprises:a collection of objects in the abstract componentclass; and a display method for displaying the collection of objectsconcurrently starting from a common time; (c) a track reference classderived from the abstract component class and defining a second subclassof the component class, wherein a track reference object, an instance ofthe track reference class, comprises:a reference to a time value withrespect to the common time and a specified one of the objects in thecollection of objects in the track group object and a duration, whereinanother one of the objects in the collection of objects contains thetrack reference object; and a display method for displaying the trackreference object by identifying one or more objects in the specifiedobject in the collection of objects in the track group objectcorresponding to the time value and duration of the track referenceobject and accessing and displaying media data associated with theidentified object.
 7. A computer system for editing a multimediacomposition, comprising:means for permitting an editor to create aninstance of a track group object including a collection of references totracks defined by components having associated media data, wherein thetracks in the track group object are displayed concurrently starting ata common time; means for permitting an editor to create an instance of atrack reference object as one component in a track in the track groupobject including a reference to another component in the track groupobject including a time value with respect to the common time and aduration, wherein the track reference object is in a different track inthe track group than the other component; and means for displaying thetrack reference object by identifying the other component in the trackgroup object specified by the track reference object and correspondingto the time value and duration of the track reference object, and fordisplaying the media data associated with the other component.